The present disclosure relates to a method for hole-filling in 3D models and a recording medium and apparatus for performing the same, and more particularly, to a method for hole-filling in 3D models that fills a hole area created while a virtual-viewpoint image is composed from single-viewpoint images, and a recording medium and apparatus for performing the same.
Due to the industrial and academic development in the three-dimensional (3D) video field, much research has been conducted on various system and display devices that provide 3D content. In addition, research has also been conducted on a system and display device that enables a user to feel virtual reality without additional devices, such as 3D glasses.
For this, a method for composing a virtual-viewpoint image through depth-image-based rendering in which the virtual-viewpoint image is composed using actual viewpoint images has been proposed. Depth-image-based rendering uses 3D warping, and thus holes are created in the virtual-viewpoint image. In this case, a small hole is created due to an estimation error of a depth value, while a large hole is created by an area exposed in an actual viewpoint image being hidden in a virtual-viewpoint image.
An interpolation method and an in-painting method have been proposed as a representative method of filling the holes. However, according to the interpolation method, geometric distortion and blurring occur along a boundary between a background area and a foreground area, and the blurring becomes more severe as the hole area increases in size.
On the other hand, the in-painting method is classified into a structure-based method and an exemplar-based method. The structure-based method has more improved properties than the interpolation method because the structure-based method effectively preserves structural elements. However, when the hole area is large, the structural elements are spread, and thus detailed information is lost. It has been demonstrated that the exemplar-based method fills the hole area with pixels copied from a neighbor patch, and achieves an enhanced result when compared to other methods including the structure-based in-painting method. Thus, research has been conducted on various modification techniques on the basis of the exemplar-based method.
Representative examples include a method of using an image registration pattern of continuous frames to update viewpoint composition, a method of temporarily merging extracted static scenes into a virtual viewpoint, etc. According to the methods, static background pixels can be successfully obtained when a foreground object has a high activity. As the activity of the foreground object decreases, the number of remaining foreground pixels increases, and thus geometric distortion and blurring still occur. Furthermore, temporal static information regarding hole pixels may be estimated from other frames through a codebook process upon viewpoint composition of a 3D video. However, the methods have a significant error in estimating a static environment that exists for a short time and are sensitive to noise positioned at a boundary of the foreground object.
Thus, more research is needed on new techniques for a method for hole-filling in 3D models based on an exemplar-based method.